This specification relates to data processing and refining image annotations.
An image search apparatus can use an image relevance model to evaluate relevance between textual user queries and images. For example, the relevance of an image to a user query can be determined, in part, based on content depicted in the image or textual content associated with the image, e.g., textual content appearing on a web page in association with the image. The image relevance model can also analyze the way in which users who submit user queries interact with images responsive to the user queries. For example, an image search system may respond to a user query with one or more image search results that each depict a respective image, which are then displayed on a user device. If the user clicks on or otherwise interacts with an image search result, then the image relevance model can infer that the underlying image of the selected image search results was an image that was responsive to the user query.